A number of cameras are usually mounted on/in a car or vehicle, which cameras are pre-calibrated, i.e. their position and orientation are exactly calculated and stored. The understanding of real-world data seen through the cameras heavily depends on these calibration values. As the car travels, due to shocks the cameras may become misaligned, so a recalibration may be required. So, camera sensors mounted on vehicles, such as self-driving cars, can be misaligned during driving.
On-line calibration or recalibration methods can correct misalignments but these methods are time-consuming, computationally heavy operations, so it is desired to run them only when camera misalignment is detected. Therefore, there is a need for an efficient camera misalignment detection, and there is also a need for a fast and simple method by which a misalignment alert can be produced.
Existing misalignment detection methods often comprise detecting external objects, like that disclosed in US 2013/0335579 A1, and/or additional markers placed on the car, wherein their detected displacement indicates misalignment of the camera. Such additional means/steps/conditions increase the complexity of prior art systems making those also more error-prone, and also require lots of computing power for feature detection.
U.S. Pat. No. 7,877,175 B2 discloses an imaging system for a vehicle, which system includes an imaging array sensor and a control. This known system processes zones or areas of interest in the captured images and adjusts processing to accommodate any misalignment of the camera that may occur during installation of the camera at the side of the vehicle. According to this document, in order to verify that the camera or imaging sensor is mounted at the vehicle (such as at an exterior portion of the vehicle) within a desired tolerance limit so as to provide the desired field of view, the camera may detect the side of the vehicle and/or the door handle or handles of the vehicle and the control may confirm that they are in the expected location in the captured images. If the control determines that the camera is not aligned or aimed at the desired location, the control may adjust the image and/or image processing to account for any such misalignment of the camera. For example, the degree of misalignment may be calculated, and the image processing may be adjusted or shifted and/or rotated to position the reference structure at the appropriate location in the captured images. This known solution does not provide a simple, robust and reliable method for generating misalignment alert, as it necessitates the implementation of complex object-detecting algorithms.
U.S. Pat. No. 9,296,337 B2 discloses a method of calibrating a vehicular camera. This solution uses a visible part of the car, the bumper, and uses only one reference point and a complex image processing algorithm for detecting misalignment. The detection of the reference point is based on an iterative search algorithm which is based on pixel intensities. This system is mainly designed for rear view cameras, and necessitates a complex image processing method, which is undesirable, and it has several special prerequisites preventing robustness. E.g. the known system takes advantage of the horizontal alignment of the car bumper; therefore it does not provide a general solution applicable to various parts of the exterior of the car. Furthermore, detecting only one point does not result in a reliable system, as e.g. dirt can appear at relevant places on both the camera and the bumper during driving, which can deteriorate or make impossible successful detection.